Capacity and Allocation across Sensory and Short-Term Memories.

Human memory consists of sensory memory (SM), short-term memory (STM), and long-term memory (LTM). SM enables a large capacity, but decays rapidly. STM has limited capacity, but lasts longer. The traditional view of these memory systems resembles a leaky hourglass, the large top and bottom portions representing the large capacities of SM and LTM, whereas the narrow portion in the middle represents the limited capacity of STM. The "leak" in the top part of the hourglass depicts the rapid decay of the contents of SM. However, recently, it was shown that major bottlenecks for motion processing exist prior to STM, and the "leaky hourglass" model was replaced by a "leaky flask" model with a narrower top part to capture bottlenecks prior to STM. The leaky flask model was based on data from one study, and the first goal of the current paper was to test if the leaky flask model would generalize by using a different set of data. The second goal of the paper was to explore various block diagram models for memory systems and determine the one best supported by the data. We expressed these block diagram models in terms of statistical mixture models and, by using the Bayesian information criterion (BIC), found that a model with four components, viz., SM, attention, STM, and guessing, provided the best fit to our data. In summary, we generalized previous findings about early qualitative and quantitative bottlenecks, as expressed in the leaky flask model and showed that a four-process model can provide a good explanation for how visual information is processed and stored in memory.

Sensory Memory Is Allocated Exclusively to the Current Event-Segment.

The Atkinson-Shiffrin modal model forms the foundation of our understanding of human memory. It consists of three stores (Sensory Memory (SM), also called iconic memory, Short-Term Memory (STM), and Long-Term Memory (LTM)), each tuned to a different time-scale. Since its inception, the STM and LTM components of the modal model have undergone significant modifications, while SM has remained largely unchanged, representing a large capacity system funneling information into STM. In the laboratory, visual memory is usually tested by presenting a brief static stimulus and, after a delay, asking observers to report some aspect of the stimulus. However, under ecological viewing conditions, our visual system receives a continuous stream of inputs, which is segmented into distinct spatio-temporal segments, called events. Events are further segmented into event-segments. Here we show that SM is not an unspecific general funnel to STM but is allocated exclusively to the current event-segment. We used a Multiple-Object Tracking (MOT) paradigm in which observers were presented with disks moving in different directions, along bi-linear trajectories, i.e., linear trajectories, with a single deviation in direction at the mid-point of each trajectory. The synchronized deviation of all of the trajectories produced an event stimulus consisting of two event-segments. Observers reported the pre-deviation or the post-deviation directions of the trajectories. By analyzing observers' responses in partial- and full-report conditions, we investigated the involvement of SM for the two event-segments. The hallmarks of SM hold only for the current event segment. As the large capacity SM stores only items involved in the current event-segment, the need for event-tagging in SM is eliminated, speeding up processing in active vision. By characterizing how memory systems are interfaced with ecological events, this new model extends the Atkinson-Shiffrin model by specifying how events are stored in the first stage of multi-store memory systems.

Can Contrast-Response Functions Indicate Visual Processing Levels?

Many visual effects are believed to be processed at several functional and anatomical levels of cortical processing. Determining if and how the levels contribute differentially to these effects is a leading problem in visual perception and visual neuroscience. We review and analyze a combination of extant psychophysical findings in the context of neurophysiological and brain-imaging results. Specifically using findings relating to visual illusions, crowding, and masking as exemplary cases, we develop a theoretical rationale for showing how relative levels of cortical processing contributing to these effects can already be deduced from the psychophysically determined functions relating respectively the illusory, crowding and masking strengths to the contrast of the illusion inducers, of the flankers producing the crowding, and of the mask. The wider implications of this rationale show how it can help to settle or clarify theoretical and interpretive inconsistencies and how it can further psychophysical, brain-recording and brain-imaging research geared to explore the relative functional and cortical levels at which conscious and unconscious processing of visual information occur. Our approach also allows us to make some specific predictions for future studies, whose results will provide empirical tests of its validity.

The reference frame for encoding and retention of motion depends on stimulus set size.

The goal of this study was to investigate the reference frames used in perceptual encoding and storage of visual motion information. In our experiments, observers viewed multiple moving objects and reported the direction of motion of a randomly selected item. Using a vector-decomposition technique, we computed performance during smooth pursuit with respect to a spatiotopic (nonretinotopic) and to a retinotopic component and compared them with performance during fixation, which served as the baseline. For the stimulus encoding stage, which precedes memory, we found that the reference frame depends on the stimulus set size. For a single moving target, the spatiotopic reference frame had the most significant contribution with some additional contribution from the retinotopic reference frame. When the number of items increased (Set Sizes 3 to 7), the spatiotopic reference frame was able to account for the performance. Finally, when the number of items became larger than 7, the distinction between reference frames vanished. We interpret this finding as a switch to a more abstract nonmetric encoding of motion direction. We found that the retinotopic reference frame was not used in memory. Taken together with other studies, our results suggest that, whereas a retinotopic reference frame may be employed for controlling eye movements, perception and memory use primarily nonretinotopic reference frames. Furthermore, the use of nonretinotopic reference frames appears to be capacity limited. In the case of complex stimuli, the visual system may use perceptual grouping in order to simplify the complexity of stimuli or resort to a nonmetric abstract coding of motion information.

Stream specificity and asymmetries in feature binding and content-addressable access in visual encoding and memory.

Human memory is content addressable-i.e., contents of the memory can be accessed using partial information about the bound features of a stored item. In this study, we used a cross-feature cuing technique to examine how the human visual system encodes, binds, and retains information about multiple stimulus features within a set of moving objects. We sought to characterize the roles of three different features (position, color, and direction of motion, the latter two of which are processed preferentially within the ventral and dorsal visual streams, respectively) in the construction and maintenance of object representations. We investigated the extent to which these features are bound together across the following processing stages: during stimulus encoding, sensory (iconic) memory, and visual short-term memory. Whereas all features examined here can serve as cues for addressing content, their effectiveness shows asymmetries and varies according to cue-report pairings and the stage of information processing and storage. Position-based indexing theories predict that position should be more effective as a cue compared to other features. While we found a privileged role for position as a cue at the stimulus-encoding stage, position was not the privileged cue at the sensory and visual short-term memory stages. Instead, the pattern that emerged from our findings is one that mirrors the parallel processing streams in the visual system. This stream-specific binding and cuing effectiveness manifests itself in all three stages of information processing examined here. Finally, we find that the Leaky Flask model proposed in our previous study is applicable to all three features.

Bottlenecks of motion processing during a visual glance: the leaky flask model.

Where do the bottlenecks for information and attention lie when our visual system processes incoming stimuli? The human visual system encodes the incoming stimulus and transfers its contents into three major memory systems with increasing time scales, viz., sensory (or iconic) memory, visual short-term memory (VSTM), and long-term memory (LTM). It is commonly believed that the major bottleneck of information processing resides in VSTM. In contrast to this view, we show major bottlenecks for motion processing prior to VSTM. In the first experiment, we examined bottlenecks at the stimulus encoding stage through a partial-report technique by delivering the cue immediately at the end of the stimulus presentation. In the second experiment, we varied the cue delay to investigate sensory memory and VSTM. Performance decayed exponentially as a function of cue delay and we used the time-constant of the exponential-decay to demarcate sensory memory from VSTM. We then decomposed performance in terms of quality and quantity measures to analyze bottlenecks along these dimensions. In terms of the quality of information, two thirds to three quarters of the motion-processing bottleneck occurs in stimulus encoding rather than memory stages. In terms of the quantity of information, the motion-processing bottleneck is distributed, with the stimulus-encoding stage accounting for one third of the bottleneck. The bottleneck for the stimulus-encoding stage is dominated by the selection compared to the filtering function of attention. We also found that the filtering function of attention is operating mainly at the sensory memory stage in a specific manner, i.e., influencing only quantity and sparing quality. These results provide a novel and more complete understanding of information processing and storage bottlenecks for motion processing.

Influence of correspondence noise and spatial scaling on the upper limit for spatial displacement in fully-coherent random-dot kinematogram stimuli.

Correspondence noise is a major factor limiting direction discrimination performance in random-dot kinematograms. In the current study we investigated the influence of correspondence noise on Dmax, which is the upper limit for the spatial displacement of the dots for which coherent motion is still perceived. Human direction discrimination performance was measured, using 2-frame kinematograms having leftward/rightward motion, over a 200-fold range of dot-densities and a four-fold range of dot displacements. From this data Dmax was estimated for the different dot densities tested. A model was proposed to evaluate the correspondence noise in the stimulus. This model summed the outputs of a set of elementary Reichardt-type local detectors that had receptive fields tiling the stimulus and were tuned to the two directions of motion in the stimulus. A key assumption of the model was that the local detectors would have the radius of their catchment areas scaled with the displacement that they were tuned to detect; the scaling factor k linking the radius to the displacement was the only free parameter in the model and a single value of k was used to fit all of the psychophysical data collected. This minimal, correspondence-noise based model was able to account for 91% of the variability in the human performance across all of the conditions tested. The results highlight the importance of correspondence noise in constraining the largest displacement that can be detected.

Misperceptions in the trajectories of objects undergoing curvilinear motion.

Trajectory perception is crucial in scene understanding and action. A variety of trajectory misperceptions have been reported in the literature. In this study, we quantify earlier observations that reported distortions in the perceived shape of bilinear trajectories and in the perceived positions of their deviation. Our results show that bilinear trajectories with deviation angles smaller than 90 deg are perceived smoothed while those with deviation angles larger than 90 degrees are perceived sharpened. The sharpening effect is weaker in magnitude than the smoothing effect. We also found a correlation between the distortion of perceived trajectories and the perceived shift of their deviation point. Finally, using a dual-task paradigm, we found that reducing attentional resources allocated to the moving target causes an increase in the perceived shift of the deviation point of the trajectory. We interpret these results in the context of interactions between motion and position systems.

High-capacity, transient retention of direction-of-motion information for multiple moving objects.

The multiple-object tracking paradigm (MOT) has been used extensively for studying dynamic visual attention, but the basic mechanisms which subserve this capability are as yet unknown. Among the unresolved issues surrounding MOT are the relative importance of motion (as opposed to positional) information and the role of various memory mechanisms. We sought to quantify the capacity and dynamics for retention of direction-of-motion information when viewing a multiple-object motion stimulus similar to those used in MOT. Observers viewed three to nine objects in random linear motion and then reported motion direction after motion ended. Using a partial-report paradigm and varying the parameters of set size and time of retention, we found evidence for two complementary memory systems, one transient with high capacity and a second sustained system with low capacity. For the transient high-capacity memory, retention capacity was equally high whether object motion lasted several seconds or a fraction of a second. Also, a graded deterioration in performance with increased set size lends support to a flexible-capacity theory of MOT.

Early age-related decline in the effective number of trajectories tracked in adult human vision.

Human performance in many visual and cognitive tasks declines with age, the rate of decline being task dependent. Here, we used a multiple-object tracking (MOT) task to provide a clear demonstration of a steep cognitive decline that begins relatively early in adult life. Stimuli consisted of 8 dots that moved along linear trajectories from left to right. At the midpoint of their trajectories, a certain number of dots, D (1, 2 or 3), deviated either clockwise or counter-clockwise by a certain magnitude (57 degrees, 38 degrees or 19 degrees); the task for observers was to identify the direction of deviation. Percent correct responses were measured for 22 observers aged 18-62 years and were converted to effective numbers of tracked trajectories (E) (S. P. Tripathy, S. Narasimhan, & B. T. Barrett, 2007). In 5 of the 7 conditions tested, there was a significant negative correlation between age and E, indicating an age-related decline in tracking ability. This decline was found to be equivalent to a mean performance drop of 16% per decade over the four decades of adulthood tested. Further analysis suggests that performance in this task starts to decline at around 30 years of age and falls off at the rate of approximately 20% every subsequent decade.

Loss of positional information when tracking multiple moving dots: the role of visual memory.

Pylyshyn, Z.W. and Storm, R.W. (1988) (Tracking multiple independent targets: Evidence for a parallel tracking mechanism. Spatial Vision, 3(3), 179-197) proposed that human observers could simultaneously track up to five dots when presented with an array of dots moving in a random manner. In contrast, Tripathy, S.P., and Barrett, B.T. (2004) (Severe loss of positional information when detecting deviations in multiple trajectories. Journal of Vision, 4(12):4, 1020-1043, http://journalofvision.org/4/14/4/, doi: 10.1167/4.12.4) showed that when a threshold paradigm was employed, observers' ability to track deviations in straight-line trajectories is severely compromised when attending to two or more dots. In this study we present a series of four experiments that investigates the role of attention and visual memory while tracking deviations in multiple trajectories using a threshold paradigm. Our stimuli consisted of several linear, non-parallel, left-to-right trajectories, each moving at the same speed. At the trajectory mid-point (reached simultaneously by all dots), one of the dots (target) deviated clockwise or counter-clockwise. The observers' task was to identify the direction of deviation. The target trajectory was cued in the second half of the trial either by disappearance of distractors at the monitor's mid-line (Experiment 1) or by means of a change in colour of the target (Experiment 2); in both cases deviation thresholds rose steeply when the number of distractor trajectories was increased from 0 (typical threshold approximately 2 degrees) to 3 (typical threshold>20 degrees). When all the trajectories were presented statically in a single frame (Experiment 3), thresholds for identifying the orientation change of the target trajectory remained relatively unchanged as the number of distractor trajectories was increased. When a temporal delay of a few hundred milliseconds was introduced between the first and second halves of trajectories (Experiment 4), deviation thresholds increased steeply. These results suggest that the persistence of trajectory-traces in visual sensory memory may play an important part in determining thresholds for detecting deviations in trajectories.

On the effective number of tracked trajectories in amblyopic human vision.

We estimated the effective number of trajectories that amblyopic observers could track with their amblyopic eyes and their non-amblyopic eyes using stimuli and methods described in S. P. Tripathy, S. Narasimhan, and B. T. Barrett (2007). The stimuli consisted of dots moving along straight-line trajectories. In Experiment 1, one of the T trajectories (the target) deviated clockwise or counterclockwise by +/-19 degrees , +/-38 degrees , or +/-76 degrees , halfway through the trajectory. In Experiment 2, D of the T trajectories deviated, all in the same direction and with the same magnitude of direction change. In both experiments, we varied T and the angle of deviation. In Experiment 2, we also varied D. Amblyopic observers reported the direction of deviation of the target trajectories and, for each eye, the effective number of tracked trajectories was estimated. This number increased systematically with increasing magnitude of deviation of the targets. On average, the effective numbers of tracked trajectories were approximately 15% smaller for the amblyopic eyes for each of the three magnitudes of deviation. A comparison with data previously published for normal eyes failed to reveal any deficit in the effective number of trajectories tracked by the non-amblyopic eyes of amblyopic observers for the current task.

Attraction of flashes to moving dots.

Motion is known to distort visual space, producing illusory mislocalizations for flashed objects. Previously, it has been shown that when a stationary bar is flashed in the proximity of a moving stimulus, the position of the flashed bar appears to be shifted in the direction of nearby motion. A model consisting of predictive projections from the sub-system that processes motion information onto the sub-system that processes position information can explain this illusory position shift of a stationary flashed bar in the direction of motion. Based on this model of motion-position interactions, we predict that the perceived position of a flashed stimulus should also be attracted towards a nearby moving stimulus. In the first experiment, observers judged the perceived vertical position of a flash with respect to two horizontally moving dots of unequal contrast. The results of this experiment were in agreement with our prediction of attraction towards the high contrast dot. We obtained similar findings when the moving dots were replaced by drifting gratings of unequal contrast. In control experiments, we found that neither attention nor eye movements can account for this illusion. We propose that the visual system uses predictive influences from the motion processing sub-system on the position processing sub-system to overcome the temporal limitations of the position processing system.

On the effective number of tracked trajectories in normal human vision.

Z. W. Pylyshyn and R. W. Storm (1988) have shown that human observers can accurately track four to five items at a time. However, when a threshold paradigm is used, observers are unable to track more than a single trajectory accurately (S. P. Tripathy & B. T. Barrett, 2004). This difference between the two studies is examined systematically using substantially suprathreshold stimuli. The stimuli consisted of one (Experiment 1) or more (Experiments 2 and 3) bilinear target trajectories embedded among several linear distractor trajectories. The target trajectories deviated clockwise (CW) or counterclockwise (CCW) (by 19 degrees, 38 degrees, or 76 degrees in Experiments 1 and 2 and by 19 degrees, 38 degrees, or 57 degrees in Experiment 3), and observers reported the direction of deviation. From the percentage of correct responses, the "effective" number of tracked trajectories was estimated for each experimental condition. The total number of trajectories in the stimulus and the number of deviating trajectories had only a small effect on the effective number of tracked trajectories; the effective number tracked was primarily influenced by the angle of deviation of the targets and ranged from four to five trajectories for a +/-76 degrees deviation to only one to two trajectories for a +/-19 degrees deviation, regardless of whether the different magnitudes of deviation were blocked (Experiment 2) or interleaved (Experiment 3). Simple hypotheses based on "averaging of orientations," "preallocation of resources," or pop-out, crowding, or masking of the target trajectories are unlikely to explain the relationship between the effective number tracked and the angle of deviation of the target trajectories. This study reconciles the difference between the studies cited above in terms of the number of trajectories that can be tracked at a time.

Is the ability to identify deviations in multiple trajectories compromised by amblyopia?

Amblyopia results in a severe loss of positional information and in the ability to accurately enumerate objects (V. Sharma, D. M. Levi, & S. A. Klein, 2000). In this study, we asked whether amblyopia also disrupts the ability to track a near-threshold change in the trajectory of a single target amongst multiple similar potential targets. In the first experiment, we examined the precision for detecting a deviation in the linear motion trajectory of a dot by measuring deviation thresholds as a function of the number of moving trajectories (T). As in normal observers, we found that in both eyes of amblyopes, threshold increases steeply as T increases from 1 to 4. Surprisingly, for T = 1-4, thresholds were essentially identical in both eyes of the amblyopes and were similar to those of normal observers. In a second experiment, we measured the precision for detecting a deviation in the orientation of a static, bilinear "trajectory" by again measuring deviation thresholds (i.e., angle discrimination) as a function of the number of oriented line "trajectories" (T). Relative to the nonamblyopic eye, amblyopes show a marked threshold elevation for a static target when T = 1. However, thresholds increased with T with approximately the same slope as in their preferred eye and in the eyes of the normal controls. We conclude that while amblyopia disrupts static angle discrimination, amblyopic dynamic deviation detection thresholds are normal or very nearly so.

Severe loss of positional information when detecting deviations in multiple trajectories.

Human observers can simultaneously track up to five targets in motion (Z. W. Pylyshyn & R. W. Storm, 1988). We examined the precision for detecting deviations in linear trajectories by measuring deviation thresholds as a function of the number of trajectories (T ). When all trajectories in the stimulus undergo the same deviation, thresholds are uninfluenced by T for T

The extent of crowding in peripheral vision does not scale with target size.

Identifying a target is more difficult when distracters are present within a zone of interaction around the target. We investigated whether the spatial extent of the zone of interaction scales with the size of the target. Our target was a letter T in one-of-four orientations. Our distracters were four squared-thetas in one-of-two orientations, presented one in each of the four cardinal directions, equidistant from the target. Target-distracter separation was varied and the proportion of correct responses at each separation was determined. From these the extent of interaction was estimated. This procedure was repeated for different target sizes spread over a 5-fold range. In each case, the contrast of the target was adjusted so that its visibility was constant across target sizes. The experiment was performed in the luminance domain (grey targets on grey background) and in the chromatic domain (green target on equiluminant grey background). In the luminance domain, target size had only a small effect on the extent of interaction; these interactions did not scale with target size. The extents of interaction for chromatic stimuli were similar to those for luminance stimuli. For a fixed target visibility, decreasing the duration of the stimulus resulted in an increase in the extent of interaction. The relevance of our findings is discussed with regard to a variety of proposed explanations for crowding. Our results are consistent with an attention-based explanation for crowding.